A Novel Economic Approach to the Evaluation of Academic Research Libraries

With the explosion of digitized information, libraries stand at the cutting edge for research universities and other institutions of higher education. At the same time, libraries are under increasing pressure to justify every expenditure. Historically the justification has involved assessing the size of a library, in various ways, and determining its rank or position among a group of peer institutions. This approach does not answer the crucial stakeholder question: does the rank in terms of cost justify the services delivered? To answer this question it is vital to know how the costs (inputs) of these libraries are related to, and determined by the services that are rendered. We apply here a new analytical tool, data envelop analysis (DEA), which can provide better understanding of the efficiency of academic research libraries. Instead of telling us the average performance among libraries being analyzed, this method, with proper model of library inputs and outputs, tells us the best practices in the peer groups as well as the technical efficiency score for each library. We report the results of a study using 1994-1995 ARL (Association of Research Libraries) statistics together with the expert judgments, in terms of constraints, to make the model more applicable to the evaluation of libraries. This method of summarizing library efficiency not only overcomes the shortcomings of size-based comparisons but also provides new information regarding specific library operation improvements. The study also has the potential to be applied in the new digital library environment. *This research is supported in part by the Council on Library and Information Resources (CLIR) under Grant #6607, and by the Alexandria Project Laboratory at SCILS, Rutgers University. 1. STATEMENT AND IMPORTANCE OF THE PROBLEM Academic libraries since the 1970’s have been suffering from stagnant budgets. Statistics from the National Center for Education Statistics and the U.S. Department of Education indicate a decrease in libraries’ percentage of education and general expenditures from 4.1% in 1970-1971 to 3.1% in 1989-1990 (Leonard, 1994). The fiscal problems faced by many institutions of higher education give us little hope that libraries overall can expect increasing funds in the foreseeable future. At the same time, libraries face a new wave of challenges. The rapid changes in information technology make it very difficult for the libraries to keep up with the increased demand for capital investment and human resources. In short, libraries are required to do more with less. Under these circumstances, university library administrators are being asked by the parent institutions to explain 1) how well the resources are utilized in terms of producing meaningful outputs, and 2) how well the library compares or competes with other libraries within some peer group. Traditionally, in the absence of better measures, the size of a library (e.g. the size of the collection) or the ranking based on some ratios such as the ones provided by the ARL (Association of Research Libraries) statistics have been used to answer above questions. However, these approaches are limited, in the sense that they fail to address the problem of efficiency and effectiveness of library operations. Consider, for example, the various rankings developed by the ARL under the “performance indicators” heading. ARL provides 30 rankings of member libraries based on various ratios taken from its annual statistics. Examples of these ratios are “professional staff as a percentage of total staff,” “library materials expenditures as a percentage of total library expenditures,” “items loaned over items borrowed,” “serials expenditures per faculty,” and “unit price of serials.” These measures are limited in two ways. First, almost all the ratios concern only about the input side of library operation: there is no attempt to contrast the inputs to the outputs. Second, since there are so many ratios, each library can pick and choose, for its own benefit, only those ratios according to which it “performed” better. Although those rankings are easy to understand and compute, they do not seem to provide solid basis of peer comparison for improvement. Nowadays the concept of benchmarking is not uncommon in higher education. Simply put, benchmarking is about learning from the “industry leaders”. In industry the most common purposes of doing benchmarking are to increase customer satisfaction and to reduce cost (Alstete, 1996; Boxwell, 1994; Keehley, 1996; Zairi, 1996). Traditional approaches to library evaluation are not particularly useful for benchmarking because they do not give any good indication of which institutions are the leaders in academic library community. Library statistics based on the central tendency (most notably mean and median) characterize libraries whose performance is mediocre, rather than exceptional. Other than the Lake Woebegone effort to place all libraries above the average, these do not help us to identify best practice. And rankings on 30 different criteria almost never project one or two institutions to the top of all lists. To adapt benchmarking to the library situation, we need a new methodology that takes into consideration multiple inputs and outputs in a way that provides useful information for peer comparison and management change. We believe that the Data Envelopment Analysis (DEA) techniques used in this report will be able to meet these requirements. Evaluation of libraries is made more difficult by the fact that the principal products of a library are services that lead to an intangible end result, the acquisition of knowledge. However, it is certainly possible to study the service activities, as service activities, and attempt to assess how well they are done. This parallels Orr (1973)’s classic distinction between how good the library is, and what good the library does, and falls under the heading of “how good the library is”. Even with this more narrow focus, there is an additional major problem: libraries deliver a complex mix of services, and consume a complex mix of resources. In this situation, some resources may be more expensive for some libraries than for others, and some services may be more important than others. The mix is expected to vary from library to library, giving precise meaning to the notion that “every library is different from other libraries”. A new approach to this problem has been developed, which combines ideas from Economics and from Operations Research, called Data Envelopment Analysis (DEA). DEA extends the idea of benchmarking to situations with many resources and services (which are usually called “inputs” and “outputs” respectively). 2. THE DATA ENVELOPMENT ANALYSIS CONCEPT: AN EXTENSION OF BENCHMARKING. 2.1 Theory DEA (Data Envelopment Analysis) grew out of a desire to measure the relative efficiencies of any number of decision making units (DMUs) with multiple inputs and outputs. The term DMU was adopted to emphasize that the technique can be applied to any level of unit within a managed organization, from an entire firm, to a plant, and down to a single operation such as the shipping room. In the work reported here the unit of analysis is the entire library. Since the DEA technique was first developed by Charnes, Cooper, and Rhodes in 1978, it has been widely applied to DMUs as diverse as banks, police departments, hospitals, tax officers, prisons, defense bases, schools and university departments. In these applications, theoretical and empirical contributions have been made by researchers in fields as diverse as operations research, economics, industrial engineering, and public policy. [Charnes, Cooper, Lewin, and Seiford (1994) is a quite technical reference. Unfortunately there are no non-technical references that we have found. Easun (1992), however, in her Section 3.1, gives a clear and readable discussion of the origins of DEA, and of several of the key concepts.] In a nutshell, DEA determines the relative efficiency of DMUs under consideration when each DMU is allowed to be best represented in its mix of both input and output. It can be argued that this technique roughly corresponds to the assertion echoed in library world that each library is unique in its mission and services. DEA has several advantages over traditional evaluation methods such as rankings, ratios, or regression analysis. First, DEA provides, for each library, one single summary score (the relative efficiency). Depending on the specific DEA model one has applied, this efficiency score tells us for example how much (proportional) reduction of inputs or (proportional) augmentation of outputs should be possible if we adopt the practices and procedures of the most efficient DMUs. Second, the possible improvements that the DEA study identifies are in reference to the observable peer groups that have similar operations. The fact that the improvement projections are based upon best practices in the comparable organizations, and not from an ideal goal set by any arbitrary means gives the DEA results face validity. As a result of these two properties, when compared to other statistical methods that attack the same problem, DEA approach can give us better information about specific ways to make improvements. The two essential ideas of DEA are these: 1. There must be a comparison set of libraries, which is the basis for all of the calculations. In this sense, DEA is an extension of the notion of benchmarking. In benchmarking one does not provide an “absolute measure” of performance, but measures each institution by comparing it to the best performance in the designated peer group. If there are five institutions, and some single measure of performance Z, whose value is (25,50,60,100,150) at the 5 institutions then the corresponding efficiency scores would be (17%, 33%, 40%, 67%, 100%). But if the top ranked library were to be removed from the peer group, the scores of the remaining libraries would become (25, 50,60,100), and the efficiencies would be (25%, 50%, 60%, 100%). 2. Each library is given “the benefit of the doubt” in determining the relative importance of the various inputs and outputs. For example, if one library consumes a lot of staff resources, it will be favored in the analysis if the effective cost of staff is set to a low value. If another library has a very large collection, it will be favored in the analysis if the effective cost of a volume is set low, and so forth. In DEA, a separate calculation is done, for each library. In this calculation the effective costs, and the values of services are adjusted to make that “focus” library look as good as possible. If it looks at least as good as any other library, from this perspective, it receives an efficiency score of 1. But, if some other library looks better than the focus library, even when the world is described in a way that is most favorable to the focus library, then it will receive an efficiency score less than 1. In DEA, there is also a (somewhat subtle) wrinkle in the definition of the benchmark. Even when there is only a single measure of service, and of resources, the situation can be represented in two dimensions. One represents the inputs, and the other represents the outputs. Proportional reduction of inputs eventually brings a library to the “efficient frontier”. But that new situation is not likely to correspond to any real library. At best it will be on a line joining two other libraries, called the “comparison set”. What this means, in the real world, is that the efficiencies of those libraries in the comparison set are such that, if we could somehow form a library which is a mixture of them, it would be a direct comparison or benchmark for the focus library. Figure 1 summarizes some of the points made earlier in a graph. Figure 1 . Envelopment surface [Adapted from Charnes et al. (1994), p.33]